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software fault-tolerance dy design diversity 

DEDK: A TOOL FOR EXPERIMENTS 


A. Avi&eais. P. Gunningbcrg 1 * . J-PJ. Kelly. R.T. Lyu, l_ Strigini 1 . PJ. Traverse 3 . *CS- Tso, 
U. Voges 4 


UCIA Computer Science Department. University of California 
Lot Angeles. CA 90024. USA 

ANtrr-et. A large number of computing systems require very high levels of reliability, availability, 
or safety. A fault-avoidance approach is not practical in many cases, mod is costly and difficult for 
software, if not impossible. One way of reducing the effects of an error introduced during the 
di-sips of a progr a m is to use multiple versions of the program, independently designed from a 
common speeficauan. If these versions are designed by indep en dent programming teams, it is to 
be rr ^ rr teA that a fault in one version will not have the same behavior as any fault in the other 
versions. Since the error s in the output of the versions wfl] be different and uncot related, it is pos- 
sible to run the versions concurrently, cross-check their results at p resp eci fied points, and mask 
errors. A DEsign Diversity experiments (DEDK) testbed has been imple m ented at UCLA to 
study the influence of common mode er ror s which can result in a failure of the entire system. The 
layered design of DEDK and ita dec isi on algorithm are described. The usage of the system and 
its application in an ongoing e xperim ent are explained. 

Key wor ds . Computer Architecture, Reliability Theory. Distributed Parameters Systems. Coding 
Errata, Fault Tolerance. 


INTRODUCTION 

A large cumber of cont emp orary computing systems 
intended far process control applications have scingent 
reliability and availability requirements. This means that 
they mast deliver the output in a timely manner with a 
high probability of being co rrect. Su ch process control 
computers with high dependability goals can be found, for 
example, in the nuclear and aerospace industries. A sim- 
ple and efficient way of reaching this d epen dability goal is 
to use an error masking approach. An error can be 
if the system is provided with enough redun- 
dancy: typically, the execution of multiple (N-fold) com- 
putations, each computation having the same objective 
(Avi2ieaisI984|. The output of each computation then is 
voted on by a more or less sophisticated dec i sion algo- 
rithm. The result is either a single output or one output 
for each comput ation channel which is within a specified, 
acceptable tolerance. 

In order to allow dependable voting on the output, only a 
minority of the computation channels may produce an 
error at a given derision point. This condition is one of 
the basic assumptions o ceded for s uc c essf ul voting. 
Furthermore, if 

- the inputs to each computation channel are con- 
sistent, 

- the outputs are voted upon (in a more or less 
sophisticated derision function), and 

- the probability of having related errors is suffi- 
ciently low. 

ihi-n. the output of the system is suffiriendy dependable. 


These assumptions are usually satisfied. The most trou- 
blesome deals with related errors. This assumption is very 
important, because, if one er ror appears simultaneously 
ia a majority of channels any d eci s i on function will pro- 
duce an incorrect result. Ther ef ore, this probability of 
common mode er ror has to be kept low. 

As long as certain design criteria are obeyed, these 
related error s are not likely to appear if they arc dee to 
internal physical faults (rupture of connection ,e. g.J, as 
these faults are likely to have an effect only on one of the 
rSsrmi-lv at a time. External faults arc more likely to pro- 
duce related errors. Ways of dealing with these errors 
arc to have the channels loosely coupled, and to use dif- 
ferent technologies for the channels. Then, an external 
fault will not strike the channels when they are in the 
same state, and they will not react in the same way. They 
are thus distinguishable. 

Another so ur c e of related errors are design errors. 
Indeed, the N copies of faulty software will all be in error 
at the same rime when provided with identical input data. 
A way co avoid these related errors is to have different 
versions of the software (and of the entire channels) 
instead of using simple copies. Thus a key attribute for 
high dependability systems appears to be div ersity- . diver- 
sity in the timing, technology, and desig n (hardware and 
software) of the different channels. 

Let us define a cross-check point (oe- point): to be the vot- 
ing point at which the different versions e x c han ge their 
results (oc- vector) for voting. The basic' assumption, that 
only a minority is in er r o r, can then also be ex pr esse d as: 
between two successive cc-points only a minority of the 
redundant channels arc likely to fail, either by procuding 
erron eous ouepue or by failing to deliver their result in 
time. Errors in the computation wjU have an cited on 
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th b cc-vccrar and arc tacrcforc detectable. The d<owa 
algorithm will compare the oc-vcoon anti will output its 
result ia form of a decision vector. 

At UCLA an ongoing research effort was started to 
investigate design diversity, the problems that can arise, 
and to estimate the efficiency in dependability improve- 
ment by the toe of design d iver sity. The main target is 
the software, and first results included the definition of 
the rrmn-pt of N -Vernon Programming Chen 1 978 ] . and 
some first generation experiments [KcUyl982], 

In order to make measurements in a multi version 
software exp erim e nt, a t est bed was needed. A basic 
requirement was to tiwniat^ rhn environments in which 
design diversity should be used. The Design Diversity 
Experiments testbed (DEDDC) has thus two asp ect s : a 
fault-tolerant computing system, and an ex p er imentation 
tooL We will develop these two aspects in this paper. 
The main layout of the DEDDC system will be given and 
the decision algorithm im p i i-tm-nteri ia DEDDC will be 
explained more closely. Finally, the use of DEDDC in 
current experiments will be described. A more complete 
description of DEDDC can be found in {Avi2ienisl98Sj. 

DEDDC AS A FAULT-TOLERANT COM- 
PUTING SYSTEM 

As stated earlier, design diversity will often be used in an 
environment with high redundancy. Therefore, the 
pwrtv-H has to be a modular, redundant syst em to allow 
different experiments, the basic requirements for DEDDC 
are the following: 

1. The different versions of the software shall be 

able to run on diff er en t hardware in order to test 
the of errocs in the hardware associated 

with any one version. Version support software, 
therefore, has to be distributed. 

2. DEDDC must run on the dis tr i buted Locus 
environment at UCLA [Walker 1983], consisting of a 
n e t w ork of about 20 VAX ll/750s, and should be 
portable to other Unix systems. 

3. A decision algorithm has to be pan of the system, 
which provides different kinds of decision functions 
for the user like bit-by-bit comparison for id ent ity. 
ynH comparison within a sprrifird tolerance. 

4. The interface for the v ersion programmer has to 
be simple, and the interface must be indep en d en t of 
the number of actual versions used. 

In order to fulfil these requirements, DEDDC was 
developed as a modular redundant system. Depending on 
the number of versions and the number of available 
machines DEDDC selects appropriate hardware. 

DEDDC itself is written in C and makes use of several 
1 1 -tk features, c. g. for string up the different pro cesse s 
and for linking the processes via pipes. Nevertheless, it 
should be possible to port DEDDC to a pure Unix system 
which provides mechanisms for mmmiiniratino between 
several CPUs. 

We use the facilities offered by the UCLA Center for 
Experimental Computer Science. Ta- machines are linked 
by an Ethernet local network. We use the Unix software 
development environment and ia intcr-process communi- 
cation features (pipes). Incus allows p m c r.sses to 


communicate «nih each other in the same way whether 
they are running oo the same machine or on different 
m a rtiinrs It is thus easy to allocate each computation 
AmtlH to a different m:»rhir\^ 

The decision algorithm implemented will be described in 
more detail later, as well as the user ■ w Doth parts 
arc designed to fulfil the above mennonei, .equiremena. 

A global view of the DEDDC system supporting N ver- 
sions is given in Fig. 1. The versions communicate with 
the different parts of DEDDC. which in turn makes use of 
the Locus operating system, and the diff eren t sites arc 
interconnected with each other via Ethernet. 



Fig. 1. Tha N sttas d OSODC 


DEDDC A LAYERED APPROACH 

DEDDC is designed as a set of hierarchically structured 
layers. Each of the sites which are selected for running 
DEDDC has as identical set of layers and entities, provid- 
ing services to ia version and the ex t er nal user. These 
layers, from top to f mm-m , are: 

- the Version Layer, 

• the Decision and Executive Layer, 

• the Synchronization Layer, 

- the Transport Layer. 

These layers arc implemented as functions, and inside a 
site, they share some data structures (sec Fig. 2). 

The Version Layer 

This layer contains the application pr ogra m version. The 
purpose of this layer is to interface the version with the 
DEDDC system. The interface function is called the 
cross-check, or ce-function since it is called by the version 
at each oc -point. Pointers to the results to be corrected 
ire sent at parameters to this function. The cc-f uacti oa 
transfers the version r ep r es entation of results into a oc- 
vector so that the DEDDC internal representation of a co- 
vector is hidden for the version progr a m. If the decision 
algorithm detects an error in the results of the version, 
the ac- function writes back the corrected results into the 
version, therefore masking er rors. 

To run on DEDDC a version must be instru m e n ted. That 
is, the version must call DEDDC at each oc cu rrenc e of a 
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Rg. 2. Tha layata on on* aNa of DEDtX 


oc-point, and pass ia results to generate the correspond- 
ing cc- vectors. We will show bow this is done in a subse- 
quent section. Currently, the available application 
languages are C and PascaL Other languages could be 
used for the versions, if the interface between this 
language and C is provided. 

The TVr-^'rm mH Fri-rrmyr T .<vr 

This layer r ecei ves oe-v ec t o rs from the versions, decides 
on the o otrect result, determines whether a version is 
faulty or not. and makes re c overy decisions. A corrected 
cc-vcctcr is forwarded to the version. All exceptions that 
cannot be bandied at lower levels are directed to this 
layer. 

The Layer has four entities, a render, a local executive, a 
decision function, and a global executive. The local execu- 
tive entity re c eives requests from the version and 
responds to the version when a decision has been taken. 
There are four different types of normal requests: inter- 
mediate ec-voctcr (on a subset of the internal state of the 
channels), output ce-vcctor. input, and version termina- 
tion. AH of them are broadcast to the other sits, and run 
through the decisi on function to ensure consistency and 
synchronization. When the version has raised an excep- 
tion from which it cannot recover, this exception is for- 
ward ed to the local execu tiv e. 

The global executive is activated when the d eci s ion func- 
tion mrfiVnfm that the result is not unanimous, or when 
some unrecoverable ex c ep tion is signaled from the version 
or some other layer. Such an ex cepti on could be disrup- 
tion of a communication connection. This global execu- 
tive provides fault diagnosis, r ec onfiguration, and fault 
r ep orting r or maintenance purposes. Basically, it has the 
same functions as the global executive found in SIFT 
[Meiliar-Smitbl982|. 

To ensure that a consistent reconfiguration de ci si on is 
taken, the global executive at each site must first get a 


consistent error report. All global executives propo s e a 
new configuration (hat is broadcast to every site and 
derided upon. The proposed configurations are voted on 
bit-by-bit which will ensure a consistent view on a new 
configuration at every correctly working site. 

The Svnchmmrarmn 1 aver 

For each physically distinc t site, this layer broadcasts the 
result from the above executive layer and eoHeets mes- 
sages with the roulB ('ec-vecor') from aH other sites. 
This layer only acrrp o messages that are both broadcast 
within a certain time interval and rhar will arrive within 
the same time interval. The collected messages are 
delivered to the decision function. A new set of results a 
accep ted when every rite has confirmed that the messa ges 
have been delivered. T his layer mahiith m>nmrmi«-a - 
tion connections b e twe e n sites. 

A p rotocol was designed to provide (be above service. 
Synchronization of the system is based on the following 
assumptions: 

- cor r ectly working v ersi ons produce exactly the 
same number of oc- vecto rs. 

• correctly working versions have similar execution 
times, Le. they *2 produce results cithin a speci- 
fied time-out interval, 

• a majority of "missing* messages does not esst at 
a majority of rites. 

- a majority of messages arc not delayed more than 
the specified time-out interval. 

Each site has both a sender and a rec e iver entity in this 
layer, which communicate with corresponding entities of 
other sites according to the protocol . The r ec e i ver entity 
mflrffl messages from the senders and it delivers them to 
the decision function. After the delivery, it sends ack- 
nowledgments back to the senders to confirm the 
delivery. When a sender entity has collected ack- 
nowledgements from all the other sites or when it has at 
least a majority of acknowledgments, it will indicate this 
to its de cision and executive layer. This indication is used 
by the layer above to restart the vet non. By using this 
indication, it is possible to ensure that all sices win start 
the new set of computations within the specified time 
interval. 

The senders and receiv ers are designed as communicating 
extended finite state machines. They respond to events 
such as commands from the local executive, messages or 
acknowledgments, and internal time-outs. State variables. 
Le. frame sequence numbers, farming predicates cm the 
state transitions are used to discriminate messages and 
acknowledgments delayed too long in the communication 
system. The specification and verification of the protocol 
is described in [Cunningbergl9SS]. 

Tie Transport-Layer 

This layer controls the communication of messages (con- 
talcing the results) between the sites. Messages are 
broadcast to all active sites. The layer makes sure that no 
message is lost, duplicated, damaged, or misaddressed, 
and it preserves the ordering of sent messages. A discon- 
nection is r ep o r ted to the layer above. 

Currently, this layer is implemented as a simple loop of 






[wnnt to pram links by UNIX intcrproecssor pipes. Since 
(bis implementation Joes not allow a site crash, a redun- 
dant int er c o nnection structure b under implementation. 
Wc are «<*r» investigating the use of nerwork -oriented 
inter-process communication, protocol to achieve more 
transportation efficiency [Cooper 1 9&t]. 

THE DECISION FUNCTION OF DEDDC 
*sp The decision function has to recognize whether the 
versions are in agreement with each other or not. The 
decision function is used for each cc-point, and each of 
these derisions is independent of the preceding ones, and 
based only an the set of cc-vecrcrrs that b transmitted by 
the synchronization layer. An agreement is achieved if at 
least a majority of versions b considered to be equivalent 
by the derision algorithm, and this value b used as an 
output. This value b also communicated to the versions 
in er r o r, so they can use it for their subsequent computa- 
tion. 

An agreement among cc-vrrrnrs means basically that 
these oC'Vcetors contain the same information, at Che level 
of abstrarmon of the user of the versions. Thb means that 
the versions (that have been desired by different pro- 
grammer teams, in different languages, that may no on 
different machines. ...) may have different ways of 
rep resenting information. The decs ion function has thus 
to errrad the meaning of the oc-vc uut s. A "bit-by-bit* 
vote can be used for much of the cc- vector since there b 
only one possible r e pre s entation of the data. Neverthe- 
less previous experiments have shown that bit-by-bit vot- 
ing can be too selective and reject semantically equivalent 
results [Kcllyl982]. 

Th erefo r e , the ce- ve rtot s b subdivided into parts, and a 
separate derision b possible for each part. The global 
dads ion vector b composed of the union of the values of 
each part. The parts can be classified in the following 
way: 

- 'matching dass". where a bit-by-bit vote b used 

(primarily for integers), 

- "cosmetic dass", where cosmetic errors are allowed 

(mainly used for character strings), 

- 'real number dass", containing real numbers 

which arc allowed to be slightly different. 

Each dass b considered separately below. 

Matching Pension 

Thb derision b applied on data that must be strictly 
equal, like binary values or integers. The comparison on 
equality b done bet w een all oc-vectors. 

Cosmetic Derision 

Cosmetic er r on are defined as errors in character strings 
like minor misspelling in a word which b to be displayed 
to the op er ator. The human would recognise the error 
and stm c orrect ly understand the word or message. If 
diverse versions arc used with a bit-by-bit vote, a 
‘cosmetically faulty* version will be declared faulty, and. 
a ccor ding to the reconfiguration policy, could be dis- 
carded. If, on the other band, the derision function can 
tolerate cosmetic erro rs , a system using design diversity 
win not be penalized in comparison to a "classical" fault- 
tolerant system. A version with cosmetic crro is need not 
be discarded. However a cosmetic err or must be db- 
anguished from a fatal er ror. 


As an example consider the integer V. it can be wnnen 
as character string '09'. V. or '_9'. which would result in 
disagreement in a bit-by-bit comparison. In contrast, if 
the word size and the munher representation arc defined, 
the comparison o( *9' as an integer would result in only 
one possible representation. Therefore numbers should 
not be represe nt ed as character strings. 

For character strings, wc have to ArriAr which misspel- 
lings to allow. In a study [ Pollock 1 983 [ misspellings 
found in several journals have been cat eg o ris ed. As the 
text of these journals has been pmressed by computer, 
the kind of misspellings in them can be ex pected to be 
representative of faults entered through a keyboard, and 
so representative of software. The Study showed that one 
misspelling occur ed for every 250 words. More than 90% 
of these misspellings can be chanctcrizerl as being 
. an omission of one charade- . 

- an insertion of one character 

• a substitution of one character by another one, 

- a transposition. of two adjacent charat.cts. 

C os meric er rors arc tolerated by Lie decision if 

they arc part of the above four cases. 

Numeric Derision 

For decisions on real numbers, two solutions are pro- 
posed: select one representative value or tolerate all 
values within a given tolerance. In the first ease, the 
representative value has to be defined and its selection 
algorithm has to be implemented, which will always result 
in an ac ce ptable solution. In the second case, the results 
of the different versions are compared with each other to 
determine whether a majority of them b close enough 
together wi thin the toler an ce. Currently, the first 
approach b implemented in DEDDC. since wc have been 
able to derive a very simple derision algorithm. Thb algcv 
rithm b summarized ir the following. 

Wc assume that an ideal value exists (IDEAL. VALUE), 
from which an allowed imprecision b defined (5_,8^). 
such that a version V. b assumed to be non faulty, if and 
only if its response (R.) b such that: 

IDEAL.VALUE - J. S R. s IDEAL-VALUE +• 8.. 

The key of the algorithm is that it can be proved that, so 
long as a majority of versions are not faulty, the median 
of all res ponses b such that: 

IDEAL. VALUE - 6_ s MEDIAN * IDEAL. VA1.UE * 8.. 

Since taking the median of numbers b very easy to do. 
we have thus a very simple way to compute a de cs ion 
value. The most diverging versions can also be detected, 
as, under the same coocfiricn as the preceding property, it 
can be proved char a version V. b faulty if 
MEDIAN + 8.' + 8. < It 
or 

R, < MEDIAN - 8_ - 8,. 

The agreement is reached in the following steps: 

- computation of the median of the skews (if the 
versions use different skews). 

- computation of the median of the responses, 

• filtration of the versions using the above medians. 
An agreement exists if a majority of versions bas not 
hrm discarded by the filter; the decision value b the 
median. 



DEDDC AS AN EXPERIMENTATION TOOL 



la "'wrap** version software tbe version of an application 
propan are aU writ ten according to the cane functional 
specification The specification must dictate not only tbe 
overall input -output transformation tbe program has to 
perform, but also which int ermediat e resales must be 
compared, and at which points in tbe execution. Tbe 
difference bet w ee n a non- redundant program and tbe 
cor r e sp on d i n g multiple version software running on 
DEDDC is miwwiwTwi for programmers. Figure 3 shows a 
prog r am written in C and its corre sp onding instrumented 
vers -on. Tbe program costumes to read tbe sjrvti m cle ri c 
and output tbe c ur r en t time until tbe user stops it. 

nil O C 

clir •cciavO: 
lose clock: 
doable f _clock: 
ek*r 

char vrcply ■ "j\i 

atatlc ctar ■ •\tt>mto la: *: .... 

gntlc char *>2 • • Do w conlmt \yfmi 
vail. CrvpljtOl — -T- II rcplrCO] — ‘j"> < 
r_eloek - tlaa(O); 
clock " f_clock: 
csla. nt"« cttaa(aclock): 
prlaW CSaMo*. at. ctta»_r«t. a21 : 
acaaf (**a*. reply): 

> 

•»it CO): 


(a) basic p rogr am 

raraloa 0 < 

char «etl«.0: 
looa clock: 
docblb f_elock; 

Ho**, t _drlft. • 2.0: 
cbir 

cbtr »r«plr ■ * 7 ^° 

•utle cur » *\t0%t« in *: 

atatlc char ■ ' 0® w eontiaaa? if/*) m l 

while (nplylol — -r- II reply COl «* -y‘) < 
r clock * ilaa(O): 

ccpolaetl . -Ski*', atjlrirt. af_clock) : 

clock ■ t_cloek: 

ctln r«t • ctloa(kcloek): 

eeomtpatCa. •SSSalS*. at. ctla«_r«t. a3) : 

cclapaco. *Sa*. reply): 

> 

r.lu-s (0) : 

> 

(b) insttumantad varrion 
Fig. X A program for displaying currant dma. 

Toe differences b et we e n tbe program and tbe version are 
as follows: 

(1) Tbe mhv of tbe main function of tbe program is 
changed from main 0 to version Q. 

(2) Tbe Cross-check function is called to d e c i d e on tbe 
dock values of different versions after the system 
dock is read. Tbe first argument specifics the cc- 
point id. Tbe second is the format which specifies 
that tbe dock value is voted on as a real number 
with a specified skew. 


(3) Instead of using primf function far standard output, 
tbe onut p ut function is used which first votes on tbe 
output values and t hen outputs them. %S specifics 
that (be string can tolerate cosmetic error. 

(4) Similarly for tbe input. rrinpm is used to input data 
from tbe standard i» pc and broadcast it to all ^ 

(3) At tbe end of tbe program, renew is used of 

exit. 


tficr Interface 

Tbe user interface of DEDDC allows users to debug the 
system is well as tbe versions, monitor tbe op er ations of 
tbe system, apply stimnli to tbe sysatn, and to collect 
empirical data d uring experi m e n t a tio n. A number of 
romm,rw4, arc available to the user for controlling tbe 

F T^ ni fiftn and defining 

Breakpoint. Tbe break rnrnmand enables tbe user to set 
breakpoints. As a breakpoint. DEDDC stops executing 
and goes into tbe user interface- where the user can enter 
to mutiny the current system stales, 

^ f pni rifMi history, or frirnnij to the system. 

Monitoring- Tbe user can examine tbe current co ntent s 
of tbe message passing through tbe transport layer by 
using tbe display «™wwi.nd- Since every message is 
logged, the user may also sp e cif y conditions in tbe display 
mmnufld to examine any m ess age logged in tbe past. 
Tbe user can also examine tbe internal system states by 
using tbe thaw command, e-g., to examine the 
breakpoints which have b ee n set. tbe res u l ts of voting, 
etc. 

Stimuli tnjfrthn. The user is allowed to inject faults to 
tbe system by changing die system states. e.g.. tbe oe- 
voctor, by using tbe modify command. 

Statistics Collection. Tbe user interface gathers em p iri cal 
data and coDeca starisrim of tbe experiments. Every 
message passing tbe transport layer is logged into a file 
with , time-stamp. This enables the user to do post- 
execution analysis or eves replay the experiment. Statis- 
tics ttv. ylijwH fmw l system Hiw 1 w i m lvr of oc-poinm 
rrrrmrA and their resuta of dcosioo are also co l lected. 

Ezcmmoa 

Several systems arc already using diverse software. c.g. 
[Andcrsonl98S, Gmdnerl979. Martin 1932. Taylor 19S1J. 
Nevertheless, it appears (in ad di tio n to tbe fact that some 
people are not yet con vin ce d of tbe usefulness of design 
diversity) that we need to know more about related 
errors . A primary goal of DEDDC is thus to evaluate 
these related err ors. By using a controlled en v ironment , it 
win be possible to "imW the errors in order to 

- trice tbe related er rors. 

- know whether the proportion of r el a te d er r o r s is 
important or not, 

- know the impact they have oo tbe dependability of 
tbe system. 

Tbe data so nk**'<nrA will be used to evaluate tbe meaning 
of design diversity and tbe architecture of future fault- 
tolerant computers. 



Another important goal at DEDOC is the evaluation of 
ipeoficatian met h od s . I nde ed, specifications are likely to 
be the "hard-core" and the ch oic e of a specification 
method has thus to be carefully evaluated. The number 
and prop ortion of related erro rs is a measure of the dfi- 
rieacy of a specification method. By efficiency, we mean 
the inherent ability of the method to reduce errors and 
other ambiguities in the resulting specifications. 

What about the cost? It has been claimed that design 
diversity was loo costly to be used. This is obviously not 
the case when the cost of a failure of the system is impor- 
tant (money or ides). Without claiming as [Gilbl974] that 
N-vers io n programming will always r edu ce programming 
cost, we consider the advantage of testing the versions in 
parallel, with DEDtX for example. Inrirnrl. the test data 
are applied to all versions tog eth e r , and no refer ence a 
nee d ed : the reference is given by the agreeing majority of 
the v er s i ons. 

To avoid ef feeing the execution tune of DEDDC. the 
experimentatioa analysis is performed off-line. During the 
execution, files are created with for each ooear r ea oe of a 
or- point . the oc-vectors of all the versions, the decision 
vector, and the diverse diagnosis and reconfiguration 
derision available in DEDDC. 


CONCLUSION 

Currently. DEDDC is completely implemented and run- 
ning. The initial number of versions eaa be 2 or more, 
and a graceful degradation occurs when a version is 
rejected as being too often faulty. An experiment is 
under design, under the management of NASA, with the 
collaboration of four universities (Un i ver si ty of Virginia, 
University of Illinois, North Carolina State University, 
and UCLA). After these experiments, some other fault- 
tolerance techniques will be tried an DEDDC (particularly 
in the domain of reconfiguration and r e co v er y). 
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